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INTRODUCTION 



“The term achievement gap is used to denote differences in the academic achievement of 
particular groups of students” (Reynolds, 2002, p. 4). The achievement gap has been an issue in 
education for many years. Lucas (as cited by D’Amico, 2001) notes that as early as 1785, 
Thomas Jefferson saw it as an important issue when he wrote his notes on Virginia. Lucas also 
points out that W.E.B. Du Bois made its elimination a cornerstone of his agenda (as cited by 
D’Amico, 2001). According to Viadero (2000), the gap has been documented since the 1960s, at 
least. 



An achievement gap is seen between upper- and lower-class students and between 
students of differing races and ethnic backgrounds. The consensus among researchers is that 
race and class are two major contributors to the gap and there are several others. Lee and 
Burkam (2002, T1 5) state, “race and ethnicity are closely associated with socioeconomic status.” 
A high proportion of African Americans are categorized as having low socioeconomic status. 
Therefore, high proportions of African American students generally score lower on standardized 
tests. Although race and low socioeconomic status do contribute to the gap, Rothman (2001- 
2002) notes even in suburban schools white students outperform their African American peers. 

The National Assessment of Educational Progress (NAEP) data from 1971-1996 shows 
that the black-white reading gap shrank by almost half and the math gap by a third (Jencks & 
Phillips, 1998). But in the 1990s the gap for fourth-grade reading and eighth-grade mathematics 
began to widen again (Haycock, 2001). According to NAEP, white fourth graders scored an 
average of 30 points higher than their African American peers in 1998. Thirty-eight percent of 
all whites scored at the proficient level or above, yet only 9% of African Americans scored at 
this level in reading. A 40-point achievement gap occurred in eighth grade mathematics between 
whites and African Americans. Thirty-four percent of the nation’s white eighth graders scored at 
the proficient level or above, yet only 5% of their African American peers scored at the same 
level in mathematics (Education Trust, 2002-2003a, 2002-2003b). 

Following the passage of the No Child Left Behind Act (Pub. L. No. 107-1 10, 2002), 
many states and districts have made increased efforts to close the achievement gap. Haycock 
(2001) states that setting standards for what students should learn is key to solving the problem. 
These standards should set a clear guide for students, parents, teachers, and administrators. 
Kentucky was one of the first states to adopt a standard-based reform over 12 years ago. This 
adoption produced the Kentucky Learner Goals and the expectation was that all children would 
meet these goals. Kentucky officials admit that all students are not meeting these goals yet, but 
“their progress is clear and compelling” (Haycock, 2001, ^ 19). In reading, 7 of the 20 top- 
performing Kentucky schools are high poverty, in math 8 of the 20, and in writing 13 of the top 
20 are high poverty (Haycock, 2001). 
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Related Research 

An Indiana school district launched a planning process in the early 1990s to help ensure 
all students succeed. The process included 600 members of the community: district staff, 
parents, and students. During a year’s time period, the group developed ten strategies to 
guarantee all students succeed. Two of those strategies included creating a student-centered 
environment and developing a core curriculum. This school system also provided tutoring 
during school hours and a reading recovery program for first graders who were not achieving on 
grade level. With these programs in place, the district’s graduation rate has improved by 8% 
since 1995 and the African American dropout rate has decreased by 7%. Also, the reading, 
language, and math achievement gaps at different grade levels have closed by 30 to 40% from 
1998 to 2001 (Fowler-Firm, 2002). 

A magnet middle school in Houston, Texas, earned an “acceptable” rating under the state 
accountability system. In this school, 98% of white students passed the state tests in 1995-96 
and only 82% of African Americans passed. Armed with this information, the school decided to 
provide additional instruction in mathematics and an after school reading and writing program to 
help close the achievement gap. After implementing these new programs, 89% percent of 
African American students in the school passed the state tests in the 2000-0 1 academic year 
(Rothman, 2001-2002). 

Researchers have noted that achievement gap differences may be due to the kinds of 
teaching that occurs in classrooms. Waxman and Huang (1997) investigated classrooms in high 
poverty and high-minority schools. They found that classrooms rated as ineffective/inefficient 
had 40% of class time where there was no interaction and students were more engaged in whole- 
class activities. In schools rated effective/efficient, students were interacting with their teachers 
about 70% of the time and more engaged in individual, student-centered learning. 

A study conducted in Canada yielded similar results (Henchey et al., 2001). These 
researchers found that teachers possessed positive attitudes and high expectations, structured 
classroom instruction, recognized the need to be accountable for performance, and understood 
that they had to be innovative if the future of the school was to be assured. There also was a 
focus on student needs, academic achievement, and other indicators of success. The schools 
provided assistance and support for students and teachers and also provided a variety and 
flexibility of structures, programs, and services (Henchey et al., 2001). Knight and Smith 
(2003) suggest that teachers who are successful in closing the achievement gap exhibit 
instructional behaviors that involve teaching for meaning as well. 

Evaluation of the KY Extended School Services Program 

The Extended School Services (ESS) program was established in 1990 as part of the 
Kentucky Education Reform Act (KERA). Designed specifically to address the needs of 
Kentucky's at-risk student population, ESS is an aggressive, proactive program for addressing 
academic problems before they become ingrained (Nesselrodt & Schaffer, 2000a, 2000b). The 
ESS program extends the school day, week, or year for students at risk of academic failure, 
providing them with additional instructional time to help them meet academic goals. Rather than 
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being an “add-on” or “stand-alone” program, ESS is designed to be an integral part of each 
school's regular academic program, thus ensuring that students receive instructional assistance in 
core content subjects in which they are performing poorly. 

In the fall of 2001, KDE contracted with a partnership of AEL and Western Kentucky 
University for a comprehensive evaluation of the ESS program during the 2001-02 school year. 
All learnings from a previous pilot-test evaluation were incorporated into AEL's evaluation 
design. Fifteen evaluation questions were assembled into five major topics: (1) identification, 
referral, and assignment of services; (2) profiles of students receiving services; (3) profiles of 
ESS programs and their implementation patterns; (4) services to students placed at risk; and (5) 
ESS implementation patterns and outcomes. 

AEL's evaluation of the ESS program utilized two major components — statewide surveys 
and site visits. These components were broken down into five main phases: statewide surveys, 
training session for site visits, fall/winter site visits, summer visits, and data analyses. The 
surveys were administered to the district and school ESS coordinators in the fall of 2001. See 
Figure 1 for a graphic portrayal of AEL’s evaluation of the ESS program. 

Purpose and Objectives 

This study is a secondary analysis of the data from the classroom observations from the 
fall/winter and summer site visits. The purpose of this study was to explore the classroom 
observation data from the ESS evaluation site visits in terms of differences across the schools 
with a minimum achievement gap and those with a large achievement gap. The objectives of 
this study were to 

• dissagregate the classroom observation data from the full ESS evaluation 
database and split those data by the minimum- and large-gap schools, 

• discover any statistically significant differences between the two achievement 
gap groups for the variables in the classroom observation instruments, and 

• compose a picture of the important environmental and instructional 
differences in minimum and large achievement gap classrooms. 
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Two Statewide Surveys 

• District ESS Coordinators 

• School ESS Coordinators 



► 



Report on 
Both 
Surveys 



Sample of 48 Kentucky Schools 




AEL CSIQ 


• 24 Elementary Schools 




Administered 


• 12 Middle Schools 


W 


to 48 Schools 


• 12 High Schools 







t 

FallAVinter Site Visits 

• 1 8 School Sites with ESS 

• Surveys: 

• ESS Teacher 

• Non-ESS Teacher 

• Parent of ESS Student 

• ESS Student 

• Interviews: 

• District ESS Coordinator 

• School ESS Coordinator 

• ESS Teacher 

• Parent of ESS Student 

• ESS Student 

• Classroom Observations: 

• Classroom Observation Form 

• QAIT Classroom Rating 

• Classroom Environmental and 
Resources Checklist 

• School and Program Description Form 

• ICCM Form 






Final 
Report of 
the ESS 
Evaluation 



Summer Site Visits 

• 6 of 1 8 Sites Above 

• Same Surveys, Interviews, 
Observations, and Forms less the 
District ESS Coordinator Interview 
and the Non-ESS Teacher Survey 




Figure 1 : Graphic Portrayal of the Evaluation 
of the Kentucky Extended School Services Program 
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METHODS 



This sections presents the samples, observation instruments, data collection procedures, 
and the data analyses processes of this study. 

Samples 

Schools. A two-stage sampling process was implemented to identify the 24 schools to 
host the evaluation site visits, of which classroom observations were one data collection process. 
In the first stage, KDE staff identified a pool of 48 schools through a six-step process that 
included reviews of student achievement data, percentage of students eligible for free or reduced- 
price meals, overall academic index score, ethnicity, school-level performance indicators such as 
novice-level readers and dropout rates, comparisons of subsets of student scores within schools, 
and geographic and demographic characteristics. This pool of schools included 24 elementary, 

12 middle, and 12 high schools. 

AEL staff completed the second stage of school sampling by securing the Johnson locale 
codes (National Center for Education Statistics, 2001) and published enrollment figures (Quality 
Education Data, 1998) for each of the 48 schools. Using a combination of building level, 
geographic, urbanicity, and enrollment information, AEL staff selected the 1 8 schools for the 
fall/winter 2001-02 site visits. These 18 schools were evenly divided on the “subsets of student 
scores within schools” (Cowley et. al., 2002). After the 18 fall/winter site visits were completed, 
AEL staff collaborated by telephone with KDE staff to identify which of the original 48 schools 
would receive the summer 2002 site visits. It was decided that it would be best for the 
evaluation to revisit some of the 18 schools selected for the site visits. Based on geography, 
building level, and general representativeness of the Kentucky ESS program, 6 of the 1 8 were 
selected for the summer site visits. 

The 48 schools in the original pool were divided into two even groups based on the 
“comparisons of subsets of student scores within schools” (Cowley et al., 2002), as stated above. 
In one group were schools where students in ESS or free/reduced-price lunch and African 
American students were all scoring within 10 points of the school average. Henceforth in this 
paper, these will be known as the “minimum-gap” schools. Table 1 displays descriptive 
information on the set of nine minimum-gap schools selected for site visits, including classroom 
observations. There were three elementary, three middle, and three high schools in this 
minimum-gap group. School enrollment ranged from 270 to 1 ,550 students and the free/reduced 
lunch eligibility of those students ranged from 14% up to 72%. Four of the nine schools were 
located in rural locales and another three were located in small town locales. 

In the second group of schools were students in ESS or free/reduced-price lunch and 
African American students who were all scoring more than 1 0 points below the school average. 
From this point in this paper, these will be known as the “large-gap” schools. Table 2 displays 
descriptive information on the nine large-gap schools selected for site visits. As above, there 
were three elementary, three middle, and three high schools in this group. School enrollment 




10 



CO 

> 

ID 

O) 



"O 

(D 

■<-» 

O 

c /3 



o 

o 

4:: 

o 

C /3 

D. 

c^J 

0 

1 

B 

S 



c 

o 

a 

o 

'S 

s 

Ul 

c 2 

hS 

(L> 

> 



O 

CO 

(L> 

Q 



(L> 

3 

c 3 

H 







t' 




















0 




















<L> 
















4 > 




2 ? 


■0 














<L> 




cd 


< 


< 










< 


>v 




J 


00 


00 








< 


GO 


H 




0 


S 


S 








C/D 




0 




<L> 

60 

C 


0 


0 










<L> 


cd 

0 

0 


u 


TD 

'S 


£/J 


c 


c 


1 


<L> 

12 

wn 


*0 

'B 


1 


<L> 


’lI 


3 




0 


0 


0 


3 




N 


(X 


0 


0 








•E 


0 




c 75 


c 






— 














cd 


2 


2 


Cd 


Cd 


Cd 


2 


2 






u. 


3 


3 


E 


E 


E 


3 


3 




ii 


D 


oC 




00 


00 


00 


Oc^ 




CB 




















0 




















0 




















:: Di 




















C 

0> u 






















(N 


vq 


0 


cq 


0 


Os 


—* 


CN 


-- 


5 CL 
C/D E 

0 


U-) 




so 


CO 


vd 


cd 


vd 




vd 


U 




















<L> 




















. 2 ^ 
qI ^ 
^ ii 




















w .ti 
3 H 


0^ 


0^ 


0^ 

CN 


0^ 

r^ 


0^ 


0^ 

0 


0^ 

CN 


00 


00 


0 > -C 








CN 


»o 


VO 




»— 




^ C 




















(L> 3 




























































<L> 

0 

C n 




vp 


vp 


vp 


Np 


Np 


Np 


Np 


Np 


-§ a 




0^ 

0 




0^ 

CN 


0^ 


0^ 

vq 


0^ 


0^ 

vq 


0^ 


C £d 
0 > Ci 
ti ^ 




VO 




vd 


vd 


'^* 


vd 






0 


On 


Os 


Os 


Os 


Os 


Os 


Os 


Os 


< 




















ea 




















•o c 




















<L> (i> 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 E 


ro 


»o 


r- 


0 


»o 


CN 


OS 


0 


VO 




CO 


‘'T 


CN 


CO 


OS 










0 p 




















oi c 




















u 




















(8 




















C/5 

<u 

*o 


(N 


(N 


PK-6 


To 

1 


Tn 


00 

1 


vn 


~oo 


00 

1 


£d 

i-~ 


1 

OS 


1 

OS 


1 

OS 


so 




so 




0 




















Lm 




Um 




Um 






Um 






< 1 > 




<U 




a> 






a> 




<l> 


E 




E 




E 






E 




E 


E 




E 




E 






E 






3 




3 




3 






3 




*H 


00 




00 




(/) 






00 






*o 




*0 




*0 






*o 




on 


c 




c 




c 






c 




> 


£d 




£d 




£d 






£d 






£d 


£d 


£d 


£d 


£d 


£d 


£d 


£d 


£d 




u. 


u. 


Cjl 


u. 


u. 


u. 


u. 


u. 


u. 






<N 


CO 




»n 


so 




00 


OS 


<u 


1 


' 


' 


i 


• 


' 


• 


• 


1 


E 

£d 

z. 


CL 


CL 


CL 


Q- 


CL 


CL 


CL 


CL 


CL 


£d 


£d 


£d 


£d 


£d 


£d 


£d 


£d 


£d 


0 


0 


0 


0 


0 


0 


0 


0 


0 




E 


E 


E 


E 


E 


E 


E 


E 


E 


0 


3 


3 


3 


3 


3 


3 


3 


3 


3 


w 

JC 


E 


E 


E 


E 


E 


E 


E 


E 


E 


0 

00 


c 


c 


c 


c 


c 


c 


c 


c 


c 












s 


s 


s 







c 

<L> 

E 

CJ 

CL 

0 ) 

Q 



c 

o 

G 

CQ 

CL 

Q 

- 

o 

3 



O 

3 

< 

u 

Q 

c 

o 

60 

c 



£6 ^ 

S o 

.2 o 

o on 
3 O 

*o 

’•S ^ 

3 C 

0 .2 

- 4 -* 

* • £6 
O « 

w 

w 

c ci 

Q w 
^ c 
zz ^ 

fN re 

S § 

£Q 

Q £6 

c Z 
o 

C 6 

= I 

•O ^ 

w Q 

£6 i.. 

3 O 

ao 

s: 
o 
S 



J 2 > 



O 

s 

s: 

I 

s 

"o 

o 

■s 



fi: 

O ^ 

c q 

■ ‘ c 



Q Co 

3 

Qb 
fN 



0 )S 

E 

o 



Q ^ 

^ 5 
> ^ 
*E *o 

'o '> 
^ § 
^ "S 

^ on 
£6 o 

= 5 

o ^ 

D > 
£2 > 

c S. 

E E 
o> o 

w 

o " 



a. w II 
E < < 
P C/3 



o 

ERIC 



11 



(/} 

> 

o 

<2 

T3 

d> 

O 

13 

00 

c/l 

'o 

O 

0 
00 

& 

a 

1 

<D 

W) 

cd 

H-) 

a 

o 

a 

'S 

E 

Ui 



(D 

> 



O 

c/) 

(D 

Q 



CN 

JJ 

3 

H 

















at 


















u 




u 














0> 




0> 














oz: 




a. 
































CO 


































O 




CO 

o 


c 




c 


c 




& 


0> 

OJC 


c 


o 










G 


u 


.£ 






o 




o 


o 




flj 




o 






G 




H 


N 


N 


Lju 


H 




o> 

w: 


<u 

OJQ 


lo 


0> 

01] 


’O 


*55 


c 

CO 

Xi 


lo 




CO 


CO 


E 

c/3 


CO 


ii 




Urn 

:d 


E 

c/3 


CQ 




































2 ^ 


















^ oC 






































to 




o 


O) 


o 


p 


p 


p 


ZZ Q, 

00 c 


rn 


’'O' 


’'O' 


rn 


rn 


to 


to 


-to- 


C 

o 


















U 


















(D 

U JO 

^ V 


















cu — 


















-A ^ 










'vO 












CN 


Os 


g 


to 


NO 


NO 




TD — 

Qi 






m 


m 










^ c 

^ 3 






















































U. 


















0) 


















o 


















C CO 


vP 


NP 


sP 


NP 


Np 


Np 


Np 


nP 


■§ S 




t/0 


NO 


o 


00 


p 


CN 


to 


C CO 








NO 


CN 


Os 


'to- 


to 




o; 


ON 


ON 


ON 


On 


00 


ON 


ON 


< 


















ca 


















"3 c 


















w a> 


o 


o 


o 


o 


O 


o 


o 


o 


"c E 


VO 








to 


NO 


o 


-to- 


3 = 


ON 




m 


NO 


to 






to 


o p 












' 1-^ 






oi 


















u 


















ra 


















CO 

d> 

-a 


<N 


1 


00 


1 


CN 


7n 


oo 

1 


"to ’ 


cO 


GTn 




m 




1 

O 


1 

o 


NO 




a 
































Urn 




<L> 














o 


<L> 


E 














E 


E 


E 














E 




3 














3 


H 


C/3 














C/3 


W 


-o 














-o 


t/5 


c 














c 


> 


CO 














CO 




CO 


CO 


CO 


CO 


CO 


CO 


CO 


CO 




u. 


u. 


u. 


u. 


u. 


u. 


u. 


U. 1 




-- 


CN 


m 




to 


NO 




oo < 




o. 


cx 


Q, 


CL 


o. 


o. 


G. 


CL 




CO 


CO 


CO 


CO 


CO 


CO 


CO 


CO 


o 


a 


O 


o 


o 


o 


o 


o 


o < 


o 


<D 


<L> 


<l> 


<L> 


<l> 


<l> 


<L> 


<l> 


c/3 


W) 


OJO 


00 


oo 


00 


00 


00 


00 




u 


U 


u 


u 


u 


u 


u 


u 




CO 


CO 


CO 


CO 


CO 


CO 


CO 


a 


















J . 



U 

d> 

w 

o 

(L> 

0£ 

C 



<l> 

0£) 



O 

ERIC 



12 



"From “School Report Cards,” by Kentucky Department of Education, 2002, Retrieved June 2, 2003, from Kentucky Department of Education website: 
http://aaps.kde.state.kv.us/report card/ 

*’* ' From QED 2001-2002 State School Guides: Kentucky (19''' ed.), by Quality Education Data, 2001, Denver, CO: Quality Education Data. 

‘ From Public Elementary/Secondary School Universe Data in Common Core of Data, by National Center for Education Statistics, 2002, Washington DC: Author. 
Available from http://www.nces.ed.gov/ccd/pubschuniv.asp 
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ranged from 370 to 1,760 students and the free/reduced lunch eligibility of those students ranged 
from 15% to 47%. There was a variety of locale types but no rural locales were included. 

Students/Classes. Students — and their classes — were selected to be observed through a 
purposeful sampling process. Following the lead from the ESS evaluation pilot test, each school 
ESS coordinator was asked to select three students to be observed in regular and ESS classes for 
approximately half of one school day by the data collector. These coordinators were asked to 
select one student who was making exceptional progress in the program, one student who was 
making typical progress in the program, and a third student who was making slower than normal 
progress in ESS. The projected number of students to be observed was 54 in the fall/winter site 
visits and 1 8 in the summer visits. 

The sampling of classes to be observed was a direct function of schedules of the students 
(three per site visit) selected by the school ESS coordinators. That is, the purposeful sampling of , 
students to be observed dictated which classes were observed. Observers were instructed to 
observe the selected students for about half of one day in their regular classes and also their ESS 
classes (before school, after school, or evening classes). Observers were instructed to follow 
each targeted student to an English/reading class and a mathematics class for sure. Due to the 
shadowing scheme for student observations, the precise number of their classes that would be 
observed was impossible to predict before the observations began, but a low-end estimate was 
108 in the fall/winter and 36 in the summer. 

In the end, because of the site selection decision for the summer visits, more visits and 
classroom observations were completed in minimum-gap schools than in large-gap schools. 
Thirteen site visits were completed in minimum-gap schools and eleven site visits were 
completed in large-gap schools. The total number of classes with some observation data was 
213; however, as is typical in these studies, not all instruments had complete data. For example, 
specific instruments may not have been completed such as the end-of-session instruments if the 
student was suddenly called out of the class for a good reason. Thus, there was some missing 
observation data, but it was not judged to be material. A total of 213 classrooms were observed 
and formed the database for this study; 193 during the fall/winter and 20 during the summer. 



Instruments 

Special Strategies Observation System. For the collection of regular classroom and 
ESS session observation data, three instruments were selected and employed in the pilot test and 
used in the evaluation. All three were developed and employed in prior research and evaluation 
studies, refined by AEL and KDE staff, and converted to a scannable format. The three 
instruments comprise the Special Strategies Observation System (SSOS), which is designed for 
use in a variety of settings to systematically collect data on essential elements of classroom 
behavior related to instruction, management, and context. The SSOS is a viable instrument for 
school effectiveness research due to its strong grounding in the current literature on effective 
teaching and its utilization of a variety of methodologies. This combination of instruments 
generates low-, moderate-, and high-inference data; this triangulation of information further 
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documents the veracity of the data collected. Each instrument that makes up the SSOS is 
described below. 

• Classroom Observation Form (COF): The COF is a combination observation system 
that is best described as a category system with low inference items and includes multiple 
coding procedures (Nesselrodt & Schaffer, 1993; Sullivan & Meehan, 1983). It is based 
on the Classroom Activity Record designed by Everston and Burry (1989) and the 
Stallings Observation System (Stallings, 1 980). The top page of the COF collects typical 
demographic information such as the school, observer, date, number of adults and 
students in class, subject being observed, and type of class (ESS or regular). The 
observations occur over 56 minutes, during which the observer switches between coding 
the entire classroom and focusing on a single student previously selected. Each of seven 
pages corresponds to eight minutes of class time. The first minute per page — the 
“snapshot” — looks at student engagement (i.e., the number of students on task, off task, 
out of the room, or waiting) and grouping strategies (i.e., whether clustered in teacher, 
aide, or student groups and type of involvement such as working alone, management, 
interaction, or socialization). The remaining seven minutes per page focus specifically on 
the target student and include coding one of 27 discrete activities for each minute. See 
Table 3 for the names of the 27 activities in the four main COF categories. 

• QAIT assessment of classroom: This instrument is best described as a moderate and 
high-inference simple coding rating device. QAIT stands for Quality of Instruction, 
Appropriate Level of Instruction, Incentive, and Use of Time. Fitting on two sheets, it 
contains 40 items grouped under the four major categories. Each item uses a Likert-type 
rating scale of 1 to 5 {Unlike this class to Like this class). This instrument was to be 
completed at the end of each observation session. 

• Classroom Environment and Resources Checklist: The Classroom Environment and 
Resources Checklist (CERC) is a low-inference simple coding sign system. Printed on 
the front of one sheet, it contains 12 classroom attributes that are coded either as present 
or not present, such as adequate lighting, use of multi-racial materials, posted 
assignments, etc. Next, 1 8 classroom resource items such as textbooks, computers, and 
worksheets are listed; observers indicate whether such resources are visible or not. If 
they are, observers indicate whether they are used during the observation. This 
instrument also was to be completed at the end of each observation session. See Table 4 
for the list of items in the CERC. 

See Figure 2 for a graphic depiction of the one-hour classroom observation process. 

These three observation instruments were tested and utilized in the pilot test by 
Nesselrodt and Schaffer (2000a, 2000b). Thus, these instruments possess face and content 
validity and have proven their utility in prior research. A high degree of inter-rater reliability 
was achieved among the data collectors, given that every participant in the training session 
passed at or above the 85% criterion of the COF coding assessment held at the conclusion of the 
training. To assess the degree of internal consistency reliability, Cronbach alpha coefficients 
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Table 3: 


Classroom Observation Form Activities by Main Categories 


Main Categories 


Activities 


Teacher-Led 


Teacher presentation of content 

Recitation or discussion 

Directions for assignments 

Small-group instruction 

Tests 

Checking 

Praising class 



Management/Organization Procedural or behavioral presentation 

Administrative routines 





Transitions 

Monitoring 


Student-Led 


Individual seatwork 

Individual seatwork at computer 

Pairs or group seatwork 

Pairs or group seatwork at computer 

Sustained writing or composition 

Sustained reading 

Hands-on learning 

Independent inquiry or research 

Student-initiated questions 


Off Task 


Teacher non-academic activity 

Waiting time 

Discipline 

Student non-academic activity 
Not occupied 
Off task 
Out of room 
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Table 4: Classroom Environment and Resources Checklist Items 



Main Category 


Items 


Classroom Environment 


Use of multi-racial materials 
Use of non-sexist materials 
Posted classroom rules 
Posted assignments 
Cheerful and inviting classroom 
Distinct activity centers 
Adequate lighting 

Comfortable ventilation/temperature 
Student work displayed 
No distracting internal noises/interruptions 
No distracting external noises/interruptions 
Open, risk-free environment 


Classroom Resources 


Textbooks 

Workbooks/activity books 
Worksheets 
Joumals/leaming logs 
Classroom library 
Reference materials 
Map and/or globe 
Games and/or puzzles 
Instructional aids/props 
Science/lab table(s) 

Classroom chalkboard 
Student-used equipment 
Overhead projector 
Television 
Computer 

Student manipulatives/hands-on materials 
Audio resources (i.e., tapes, CDs, players) 
Video resources (i.e., tapes, discs, players) 
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Figure 2: Graphic Depiction of the One-Hour Classroom Observation Process 
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were computed for this administration of the COF and QAIT instruments, excluding 
demographic-type items. This procedure was not appropriate for the CERC instrument, given its 
lack of variance in response options of either selected or not selected. For the COF instrument, 
this administration of the grouping strategy items resulted in an alpha coefficient of .54; for the 
student engagement items, a .82. For the QAIT instrument, this administration of all items 
resulted in a coefficient of .94; by subscale, the coefficients were .91 for quality of instruction, 

.74 for appropriate level of instruction, .88 for incentive, and .80 for use of time. 

Data Collection 

Class observations were completed as one component of data collection conducted during 
the site visits. The 18 fall/winter site visits were completed by the end of February 2002 and the 
6 summer visits all were conducted during June 2002. 

Procedurally, the pair of data collectors/observers asked the school ESS coordinators to 
select the three students to be shadowed as one of the first tasks in the site visit. Alternates were 
sought at this point, anticipating some replacement due to absences, etc. Once the students were 
selected, their daily schedules for the days the observers were onsite were obtained. With the 
students’ daily schedules as input, the pair of data collectors decided who was going to observe 
each student and when. The COF segments were completed during the observation period and 
the QAIT and CERC instruments were completed at the end of the observation period. 

At the conclusion of each site visit, data collectors returned all materials, including the 
observation instruments, to AEL. Each set of returned site visit materials was logged in and 
checked for completeness in preparation for data analyses. 

Data Analyses 

After the fall/winter site visits were completed, AEL staff designed data entry templates 
using Remark scanning software. SSOS data were scanned by school; data files were then 
cleaned and exported to SPSS for statistical analyses. School files were merged into one master 
file before analyses began. 

COF data were averaged across the number of eight-minute intervals per each 
observation. Percentages of time for the classroom snapshots and target student activities were 
calculated for both the minimum- and large-gap groups. Data were analyzed using the 27 
individual categories and by grouping these into four main categories: teacher-led, student-led, 
management/organization, and off-task. Student engagement data and time spent by the target 
student in the four main instructional categories also were analyzed for the minimum- and large- 
gap groups. Independent Hests were computed on all these COF variables and, as appropriate, 
effect sizes were calculated for significant differences. The alpha level was set at .05. 

QAIT data were analyzed by the 40 items individually and by the four main subscales 
composed of their items. Descriptive statistics were used to describe results for both minimum- 
and large-gap groups. Next, independent t tests were conducted to determine if a statistically 
significant difference existed between the achievement gap groups for the 40 individual items 
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and each of four main categories: quality of instruction, appropriate level of instruction, 
incentive, and use of time. Again, alpha was set at .05 and effect sizes were calculated for 
significant differences. 

CERC data were analyzed by calculating crosstab frequency percentages showing 
whether the classroom attributes were present in the minimum- and large-gap groups. As well, 
frequency percentages were calculated to show whether various classroom resources were visible 
and used during the observations. Next, chi-square tests of independence were computed on the 
classroom environment and resources variables by the minimum- and large-gap groups, and 
Cramer’s V value was used for the effect size. 




19 



15 



FINDINGS 



This section presents the findings from the classroom observations in the minimum- and 
large-gap high-performing schools in Kentucky. These findings are presented first by classroom 
environment and resources, followed by instructional and teaching results. Only statistically 
significant differences between the two achievement gap groups for each instrument are 
presented. 

Classroom Environment and Resources Results 

Table 5 displays the statistical information for the significant chi-squares for classroom 
environment items by the minimum- or large-gap groups. Of the 12 classroom environment 
issues, significant differences were found on four items. Those four environmental items were use 
of multi-racial materials; cheerful and inviting classroom; no distracting external noises/ 
interruptions; and open, risk-free environment. As shown in the last column in Table 5, in each 
case, the group with the larger number of times the item was selected than expected was the 
minimum-gap group. That is, the significant differences for all four classroom environment items 
favored the minimum-gap group over the large-gap group. However, in terms of practical 
importance of the significant differences, the Cramer’s V values were all .25 or less, or small in 
Cohen’s (1977) suggested qualitative scheme for interpreting effect sizes. 

Table 6 displays the statistical information for the significant chi-squares for classroom 
resource items by the minimum- or large-gap groups. Of the 1 8 classroom resource items and the 
dual checkoffs of visible or used during the observation, significant differences were found for 
three items: textbooks, workbooks/activity books, and overhead projector. For two of the 
resources (workbooks/activity books and overhead projector), the differences were that they were 
visible in the classroom while for textbooks, the difference was that they were observed being 
used. For two of the resources (textbooks and workbooks/activity books), the difference in times 
observed favored the minimum-gap group; for the overhead projector, the difference favored the 
large-gap group. As above, the effect sizes all were under .25, or less than small in Cohen’s 
scheme. 

QAIT Instrument Results 

The QAIT classroom rating results are presented first by the individual items making up 
the four subscales and second by the subscale level. 

Table 7 displays the QAIT rating results for the significant differences of items in the 
quality of instruction subscale by the minimum- and large-gap groups. Of the 12 rating items in 
the quality of instruction subscale, significant differences were found on the nine items listed in 
the left column. All of the items’ differences were in favor of the minimum-gap group, which 
had only one item mean rating under 3.00 on the 5-point rating scale. The differences in the 
mean ratings ranged from .38 (reminds students of previously learned materials) to .87 (teacher 
uses an appropriate pace to cover content). The probabilities ranged from .000 up to .048. 
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Table 5: Statistical Information for Significant Chi-Squares for Classroom 
Environment Items by Minimum- or Large-Gap Groups 



CERC Item 


Degrees of 
Freedom 


Chi-Square 

Value* 


Cramer’s 
F Value 


Group with 
Larger Number 
Than Expected 


Use of multi-racial materials 


1 


8.38 


.20 


Minimum 


Cheerful and inviting classroom 


1 


4.53 


.15 


Minimum 


No distracting external noises/ 
interruptions 


1 


6.19 


.17 


Minimum 


Open, risk-free environment 


1 


12.63 


.25 


Minimum 



* p <.05 



Table 6: Statistical Information for Significant Chi-Squares for Classroom Resource Items 
Either Visible or Used by Minimum- or Large-Gap Groups 



CERC Item 


Visble or 
Used 


Degrees of 
Freedom 


Chi- 

Square 

Value* 


Cramer’s 
F Value 


Group with 
Larger Number 
Than Expected 


Textbooks 


Used 


1 


7.07 


.18 


Minimum 


Workbooks/ 












activity books 


Visible 


1 


8.31 


.20 


Minimum 


Overhead projector 


Visible 


1 


5.52 


.16 


Large 



*p <.05 
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Table 7: Significant Differences in Mean Ratings in Quality of Instruction Subscale Items 

by Minimum- and Large-Gap Groups 



Q of I Subscale 


Minimum Gap 


Larg 


e Gap 


df 


t 




Diff. 


Cohen’s 


Item 


Mean 


Std. Dev. 


Mean 


Std. Dev. 


P 


d 


Organizes information in 
an orderly way 


4.03 


1.28 


3.45 


1.40 


203 


3.07 


.002 


.58 


.43 


Notes transitions to new 
topics 


3.43 


1.52 


2.80 


1.53 


202 


2.95 


.004 


.63 


.41 


Frequently restates 
essential principles 


3.78 


1.49 


3.17 


1.52 


202 


2.87 


.005 


.61 


.40 


Uses devices such as 
advanced organizers 


2.85 


1.72 


2.24 


1.40 


202 


2.15 


.006 


.60 


.39 


Reminds students of 
previously learned 
materials 


3.77 


1.41 


3.39 


1.31 


204 


1.99 


.048 


.38 


.28 


The teacher shows a sense 
of humor 


3.38 


1.45 


2.83 


1.51 


206 


2.67 


.008 


.55 


.37 


Conducts formal and/or 
informal assessments 


3.95 


1.27 


3.19 


1.52 


204 


3.89 


.000 


.76 


.54 


Provides immediate and 
corrective feedback 


4.10 


1.23 


3.48 


1.49 


205 


3.27 


.001 


.62 


.46 


Teacher uses an 
appropriate pace to cover 
content 


4.26 


1.03 


3.39 


1.47 


205 


4.88 


.000 


.87 


.70 
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Effect sizes, as measured by Cohen’s value (1988) ranged from .28 to .70. In terms of 
Cohen’s qualitative scheme for these effect sizes, all were somewhere between small and large 
(two were medium), indicating that all of them did possess some practical importance. 

Table 8 presents the QAIT rating results for the significant differences of items in both 
the appropriate level of instruction and incentive subscales. Of the seven rating items in the 
appropriate level of instruction subscale, only one significant difference was discovered and it 
was for the “Uses individualized instruction” item. With a mean rating of 2.69 on the 5-point 
scale, the minimum-gap group was .51 points higher than the large-gap group. The probability 
was .023 and the Cohen’s d value was .32, which is small in qualitative terms. Of the 18 items 
in the incentive subscale, significant differences were found on three items, all favoring the 
minimum-gap group. The mean scores for two of those items for the minimum-gap group were 
above 3.0 (communicating high expectations and efforts by the students lead to success) and 
one was under 3.0 (relating topics to students lives). The differences in the means ranged from 
.42 to .57 and the probabilities ranged from .008 to .01 1 (tie). The three Cohen’s d values were 
similar at .35, .36, and .37, respectively, indicating some practical importance. 

Table 9 shows the QAIT observation rating results for the use of time subscale items. 
This subscale consists of three items and significant differences were found on all three items in 
favor of the minimum-gap group. Although the mean ratings for both groups were higher than 
the items on the other subscales, the significant differences in those ratings were .33 (students 
attend to lessons), .47 (necessary time is allocated for instruction), and .48 (the teacher uses 
effective management). The probabilities ranged from .005 to .024. The Cohen’s d values 
were .32, .39, and .40, indicating that the differences had some practical importance. 

Another way to present the QAIT observation results is at the subscale level with all the 
items’ ratings averaged. Table 10 displays the QAIT subscale results. Two subscales (quality 
of instruction and use of time) yielded significant differences, both favoring the minimum-gap 
group. The difference in mean scores was .51 for quality of instruction and .43 for use of time. 
The probabilities were .51 and .43, respectively, and their Cohen’s d values were similar at .51 
and .45. The qualitative descriptor for the quality of instruction d value is medium and is small 
for use of time. These effect sizes indicate that there was some real, practical importance to the 
statistically significant differences on these two subscales. 

Classroom Observation Form Results 

Results from the COF observations are presented in terms of the number of minutes. 
There were 27 individual categories of activities, four main categories of those 27 activities 
collapsed, four major groupings of subjects in the classrooms, and four categories of off-task/on- 
task behaviors coded. The 27 individual activities and four collapsed categories were coded in 
terms of what the selected students were doing, while the latter 8 categories were snapshots of 
the full classroom. Additionally, the total number of minutes observed for the students was 
computed. Table 1 1 displays the significant differences in observed minutes of instruction by 
the minimum- and large-gap groups. Of the 40 possible COF categories, significant differences 
across groups were found on just three of them (administrative routines, total number of 
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Table 8: Significant Differences in Mean Ratings in Appropriate Level of Instruction and 
Incentive Subscale Items by Minimum- and Large-Gap Groups 



A.L. of I. and Incentive 


Minimum Gap 


Large Gap 


df 


t 




Diff. 


Cohen’s 


Subscale Items 


Mean Std. Dev. 


Mean Std. Dev. 


P 


d 



Appropriate Level of Instruction Subscale 



Uses individualized 
instruction 



2.69 



1.64 



2.18 



1.55 



207 



2.30 



.023 



.51 



.32 



Incentive Subscale 



Relating topics to 
students lives 


2.70 


1.59 


2.14 


1.47 


206 


2.67 


.008 


.57 


.37 


Communicating high 
expectations 


3.34 


1.46 


2.82 


1.49 


207 


2.57 


.011 


.52 


.35 


Efforts by the students 
lead to success 


3.89 


1.09 


3.47 


1.27 


205 


2.56 


.011 


.42 


.36 



Table 9: Significant Differences in Mean Ratings in Use of Time Subscale Items 
by Minimum- and Large-Gap Groups 



Use of Time 


Minimum Gap 


Large Gap 


df 


t 




Diff. 


Cohen’s 


Subscale Items 


Mean 


Std. Dev. 


Mean 


Std. Dev. 


P 


d 


Necessary time is allocated 
for instruction 


4.18 


1.08 


3.81 


1.32 


205 


2.77 


.006 


.47 


.39 


The teacher uses effective 
management 


4.30 


1.02 


3.82 


1.37 


205 


2.84 


.005 


.48 


.40 


Students attend to lessons 


4.26 


0.90 


3.93 


1.20 


205 


2.27 


.024 


.33 


.32 



Table 10: Differences in Mean Ratings on Four Subscales of the QAIT Instrument 
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by Minimum- and Large-Gap Groups 



QAIT Subscale 


Minimum Gap 


Large Gap 


df 


t 




Diff. 


Cohen’s 


Name 


Mean 


Std. Dev. 


Mean 


Std. Dev. 


P 


d 


Quality of Instruction 


3.64 


0.96 


3.13 


1.03 


208 


3.69 


.000 


.51 


.51 


Appropriate Level of 
Instruction 


2.44 


1.02 


2.34 


0.93 


207 


0.74 


.460 


.10 


.10 


Incentive 


2.43 


0.71 


2.27 


0.75 


208 


1.63 


.105 


.17 


.23 


Use of Time 


4.28 


0.75 


3.85 


1.14 


205 


3.15 


.002 


.43 


.45 



Table 11; Significant Differences in Observed Minutes of Instruction 
by Minimum- and Large-Gap Groups 



Classroom Observation 


Minimum Gap 


Large Gap 


df 


t 




Diff. 


Cohen’s 


Form Category 


Mean 


Std. Dev. 


Mean 


Std. Dev. 


P 


d 


Administrative routines 


1.96 


1.52 


3.63 


2.34 


41 


- 2.84 


.007 


- 1.67 


1 

bo 


Total number of 
observed minutes 


47.14 


8.71 


42.73 


10.16 


211 


3.38 


.001 


4.41 


.47 


Student-led activities 


22.06 


13.44 


18.06 


12.27 


164 


2.01 


.046 


4.00 


.31 
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observed minutes, and collapsed student-led activities). In each case, the difference favored the 
minimum-gap group. For example, the large-gap group was observed spending an average of 
1 .67 minutes more on administrative routines than the minimum-gap group. This difference in 
minutes was significant at the .007 level. Further, Cohen’s d value for this difference was -.87, 
large in Cohen’s (1988) scheme, indicating much practical importance to that difference. With 
respect to the total number of minutes observed, the difference was 4.41 minutes more for the 
minimum-gap group and the probability of that difference was .001 . Cohen’s d value for total 
minutes was .47, quite nearly at the medium level. Thus, the difference in observed minutes was 
rather important in a practical sense, in addition to the statistical sense. Finally, there was a 
significant difference in the number of minutes of student-led activities. There was a four- 
minute difference in student-led activities in favor of the minimum-gap group for the 1 64 classes 
where this code was observed. The probability of obtaining this difference was .046 and the 
Cohen’s value was .31, in the small category. Thus, we can say that the difference in minutes 
of student-led activities in favor of the minimum-gap group had some practical importance in 
addition to its statistical significance. 
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CONCLUSIONS AND RECOMMENDATIONS 



Based on the findings from this study, certain conclusions and recommendations are 
warranted. First, it should be noted that the study was of the descriptive information differences 
between minimum and large achievement gap schools based on systematic observations of 
classrooms in those schools. Thus, causal inferences are not being claimed. Nonetheless, the 
observed differences are real. This section is organized by several main topics. 

Classroom Time 

The results of this study show that there were important differences in the use of time in 
the classrooms of minimum- and large-gap schools. Classroom time was used more efficiently 
and effectively in the minimum-gap schools. Teachers in the large-gap schools spent more time 
on administrative routines in their classrooms. Teachers in minimum-gap schools were observed 
providing instruction to students for more minutes than in large-gap schools. Too, we learned 
that minimum-gap teachers deployed more time for student-led activities than their counterparts. 
Teachers in minimum-gap schools were observed using effective management techniques and 
allocating necessary time for instruction. Also, students in those classes were observed attending 
to those lessons better than their counterparts. Finally, the minimum-gap teachers used an 
appropriate pace in their classrooms to cover the content. 

Although all the schools in this study were identified as high performing in terms of 
overall academic index scores, the differences in time use in their classrooms is important. The 
minimum-gap schools should know this as feedback to one of the areas in which they are doing 
well, and work to continue to do well. But more importantly, the large-gap schools should learn 
of these results because, on the basis of the overall academic index score, they may feel they 
really are high-performing schools in all areas. In reality, when compared to similar schools also 
identified as being high performing but also with subsets of students, the large-gap schools could 
do much better with the use of their classroom time. Although the actual number of minutes 
difference between the two groups of schools appears small, when multiplied by the number of 
hours per day, days per week, weeks per month, and months per school year, the difference in 
minutes is huge. For example, in terms of classroom time spent on administrative routines, 
observed teachers in the large-gap schools, on average, spent 24.35 hours more over one school 
year (1.67 x 5 periods = 8.35/day x 175 days = 1,461 minutes). In the case of student-led 
activities, observed teachers in the minimum-gap schools spent, on average, 58.33 more hours 
over the school year. 

In addition to informing schools about the differences in the use of classroom time, one 
recommendation would be to disseminate information of efficient use of time to educators in 
these schools. For example, there are research-based professional development programs to 
improve teachers’ classroom management and organization skills in order to improve their use of 
instructional time. These programs, or even demonstrations of some efficient use of time 
techniques, could be demonstrated to school staff. 
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Instructional Resources 

While the minimum- and large-gap classrooms were observed in regard to the presence 
and/or use of a wide variety of resources, only a few significant differences were found in this 
investigation. Textbooks and the use of multi-racial materials were discovered more in the 
minimum-gap schools. And, while the use of textbooks is understandable, the use of multi-racial 
materials is a major discovery of this study. One would expect that all high-performing schools 
had similar uses of multi-racial materials, but this study revealed more use of such materials in 
the minimum-gap schools. Unfortunately, the observation system did not pinpoint specifically 
what type of multi-racial materials were used in the classrooms, but their use was noted. 
Expansion of the use of multi-racial materials seems to be an important recommendation 
emanating from this study. 

The other differences in classroom resources are interesting, but puzzling. Here we are 
referring to the differences in the two items that were “visible” in the minimum- and large-gap 
schools: workbooks/activity books and overhead projector. The former favored the minimum- 
gap schools while the latter favored the large-gap schools. As noted above, we found that 
minimum-gap school classrooms used textbooks more than their counterparts, but they had more 
workbooks/activity books visible — not used — ^than their counterparts. So, one inference is that 
merely having workbooks/activity books is not as important as actually using textbooks in 
classrooms. 

Classroom Climate 

Another one of the interesting findings of this study is the “picture” of the classroom 
differences between the minimum- and large-gap schools. Here we are talking about the climate 
or tone of the classrooms that emerged from the observations. The climate in the minimum-gap 
school classrooms was more cheerful and inviting, open and risk-free, and had less distracting 
external noises/interruptions. Also, minimum-gap school classroom teachers showed a sense of 
humor more than their counterparts in large-gap schools. Certainly, taken together, these 
variables paint a very positive and inviting picture of the minimum-gap classrooms. The 
inference drawn from this picture of these classrooms is that it is worth the effort to seek to attain 
and maintain such a climate or tone for the classrooms. The implications for doing so may be 
more difficult to achieve, however. The recommendation here is transmit this finding to those in 
charge of supervising classroom teachers in the hopes that much of it can be advocated in 
classrooms. We are not sure that teaching educators how to have a sense of humor in their 
teaching is viable as a professional development target. However, we do think that professional 
development sessions can, and probably should, be on the topics of creating cheerful and inviting 
classrooms and maintaining open, risk-free environments. Too, the negative effects of 
distractions and interruptions to classrooms has been well-documented and researched (e.g., 
Sullivan & Meehan, 1983). 




28 



t 



24 



Expectations and Feedback 

Clearly, the roles of expectations and feedback to students differed across the two 
achievement gap groups. Teachers in the minimum-gap schools communicated high 
expectations to their students, conducted formal and/or informal assessments of their students, 
and also provided immediate and corrective feedback to students. These actions were 
significantly less evident in the large-gap schools. These are basic teaching behaviors that have 
been part of the effective instructional practices scene for many years. It is rather disappointing 
to discover these particular differences, especially in schools that have been identified as high 
performing. A natural recommendation is that professional development be provided to staff in 
the large-gap schools on the topics of high expectations for all students and providing 
assessments and feedback to students. We know that there has been much progress in Kentucky 
schools on these topics since the enactment of KERA, but this study showed that there is a way 
to go yet. 

Quality of Instruction 

One of the most surprising results of this study was differences in the quality of 
instruction between the minimum- and large-gap schools. Not only was the quality of instruction 
subscale itself different in a meaningful way between the two achievement gap groups, but so too 
were many of the items in the subscale. This was rather surprising and also important. The 
principles of effective instruction have been well known for many years and the expectation that 
they have been integral components of teacher preparation programs is not unwarranted. To 
discover differences on the basic instructional principles is disheartening on the one hand, but 
illuminating on the other. The illumination resides in that these instructional techniques can be 
improved. There are models, there is research, there are programs to help teachers learn to use 
these principles and skills in their teaching. 

The recommendation is that some consideration be given to professional development 
sessions for teachers in schools with large achievement gaps on many of the basic principles of 
effective instruction. Such topics include organizing information orderly, transitions to new 
topics, restating essential principles, advanced organizers, and reminding students of previously- 
learned materials. 

Level of Instruction 

Finally, this study uncovered differences in the level of instruction between the 
minimum- and large-gap schools. Although not as far-reaching or pervasive as the differences in 
the quality of instruction area, there were some important differences between the two groups. 

In the minimum-gap schools, the classrooms were observed relating topics to students’ lives 
more than their counterparts. Also, it was observed that the efforts by the students led to success. 
Again, these are crucial instructional techniques that have been known for years. Obviously, 
some teachers in some schools do better at it than others. This needs to be communicated and 
discussed among school staff The recommendation is that the differences on these topics be 
reduced in the schools. 




29 



25 



Overall 



In sum, this study provides evidence of the disparity that exists even between schools that are 
classified as high performing. While the schools in this study were performing at a high level 
overall, further investigation did illuminate the subtle classroom differences between those 
schools serving all students well and those that were not as successful with specific subgroups of 

students. 
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